_optional_components was introduced in diffusers v0.22.0 https://github.com/huggingface/diffusers/blob/v0.22.0/src/diffusers/pipelines/stable_diffusion_xl/pipeline_stable_diffusion_xl.py#L144, so we would need to upgrade DIFFUSERS_MINIMUM_VERSION as well to avoid any potential issue when loading a SDXL model (which don't have optional components)

DIFFUSERS_MINIMUM_VERSION was upgraded

Could you extend on why this is needed ? To my knowledge this shouldn't be needed (both pipelines can be correctly loaded using the current _DIFFUSERS_TASKS_TO_MODEL_LOADERS

also at the moment inferred task for diffusers models will either be "stable-diffusion-xl" or "stable-diffusion"

This is necessary to correctly determine the model class from task. Hybrid quantization supports the stable-diffusion-xl task, but doesn't support the stable-diffusion-xl-refiner. Without these changes, it is not possible to apply hybrid quantization via CLI.

Not sure that we need it in the TaskManager, from what I see it's only needed to get the class to load the model in the optimum-intel CLI for diffusers models, could we move it in optimum-intel directly instead ? We could get this information by combining the task (text-to-image, image-to-image, inpaint, ..) and the architecture (SD, SDXL, LCMs, ..) using the model's config

Are you suggesting to remove only the latent-consistency from _DIFFUSERS_TASKS_TO_MODEL_LOADERS?

As far as I understand, there are no text-to-image, image-to-image, inpaint,.. tasks in optimum. Should I create these tasks only for optimum-intel instead of stable-diffusion, stable-diffusion-xl, stable-diffusion-xl-refiner and latent-consistency? In this case, we cann't use TasksManager.infer_task_from_model for diffusion models.